{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "afd55886-5f5b-4794-838e-ef8179fb0394",
   "metadata": {},
   "source": [
    "##### **** These pip installs need to be adapted to use the appropriate release level. Alternatively, The venv running the jupyter lab could be pre-configured with a requirement file that includes the right release. Example for transform developers working from git clone:\n",
    "```\n",
    "make venv\n",
    "source venv/bin/activate && pip install jupyterlab\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c45c3c6-e4d7-4e61-8de6-32d61f2ce695",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "## This is here as a reference only\n",
    "# Users and application developers must use the right tag for the latest from pypi\n",
    "#!pip install data-prep-toolkit\n",
    "#!pip install data-prep-toolkit-transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebf1f782-0e61-485c-8670-81066beb734c",
   "metadata": {},
   "source": [
    "##### ***** Import required Classes and modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bae63d15-4ce5-4f2a-a917-0f3161e9dd73",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dpk_readability.runtime import Readability"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7234563c-2924-4150-8a31-4aec98c1bf33",
   "metadata": {},
   "source": [
    "##### ***** Setup runtime parameters for this transform\n",
    "We will only provide a description for the parameters used in this example. For a complete list of parameters, please refer to the README.md for this transform:\n",
    "|parameter:type | value | description |\n",
    "|-|-|-|\n",
    "| input_folder:str | \\${PWD}/test-data/input/ | folder that contains the input parquet files for the extreme tokenized algorithm |\n",
    "| output_folder:str | \\${PWD}/output/ | folder that contains the all the intermediate results and the output parquet files for the extreme tokenized algorithm |\n",
    "| readability_contents_column_name:str | text | name of the column that stores document text |\n",
    "| readability_score_list:Union[str, list[str]] | mcalpine_eflaw_textstat | list of readability scores or a single readability scores to be computed by the transform |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a54a78e9-d78b-4aeb-ac2b-806070a2dec0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "19:29:24 INFO - Readability parameters are : {'readability_contents_column_name': 'contents', 'readability_score_list': ['mcalpine_eflaw_textstat']}\n",
      "19:29:24 INFO - pipeline id pipeline_id\n",
      "19:29:24 INFO - code location None\n",
      "19:29:24 INFO - data factory data_ is using local data access: input_folder - test-data/input output_folder - output\n",
      "19:29:24 INFO - data factory data_ max_files -1, n_sample -1\n",
      "19:29:24 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "19:29:24 INFO - orchestrator readability started at 2025-02-10 19:29:24\n",
      "19:29:24 INFO - Number of files is 1, source profile {'max_file_size': 0.014194488525390625, 'min_file_size': 0.014194488525390625, 'total_file_size': 0.014194488525390625}\n",
      "19:29:25 INFO - Completed 1 files (100.0%) in 0.006 min\n",
      "19:29:25 INFO - Done processing 1 files, waiting for flush() completion.\n",
      "19:29:25 INFO - done flushing in 0.0 sec\n",
      "19:29:25 INFO - Completed execution in 0.006 min, execution result 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Readability(\n",
    "    input_folder=\"test-data/input\",\n",
    "    output_folder=\"output\",\n",
    "    readability_contents_column_name=\"contents\",\n",
    "    readability_score_list=[\"mcalpine_eflaw_textstat\"],\n",
    ").transform()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3df5adf-4717-4a03-864d-9151cd3f134b",
   "metadata": {},
   "source": [
    "##### **** The specified folder will include the transformed parquet files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7276fe84-6512-4605-ab65-747351e13a7c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['output/readability-test.parquet', 'output/metadata.json']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "glob.glob(\"output/*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d30489d9-fc98-423e-90a8-e8f372787e88",
   "metadata": {},
   "source": [
    "***** print the input data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5b22234f-f7a1-4b92-b2ac-376b2545abce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (2, 2)\n",
      "┌─────────────────────────────────────────────────┬────────────────────────────────────────────────┐\n",
      "│ contents                                        ┆ id                                             │\n",
      "│ ---                                             ┆ ---                                            │\n",
      "│ str                                             ┆ str                                            │\n",
      "╞═════════════════════════════════════════════════╪════════════════════════════════════════════════╡\n",
      "│ Six Sigma Tips                                  ┆ <urn:uuid:01d4d837-d379-4466-8715-3d1edda758c1 │\n",
      "│ These Six Sigma tips help you achieve high      ┆ >                                              │\n",
      "│ quality every time, the desired objective of    ┆                                                │\n",
      "│ most every business. However to get the desire  ┆                                                │\n",
      "│ of high quality creations turn into reality     ┆                                                │\n",
      "│ calls to extend yourself for some extra miles.  ┆                                                │\n",
      "│ Six Sigma is a success proven business          ┆                                                │\n",
      "│ management scheme that can assist you to        ┆                                                │\n",
      "│ unleash new horizons of business success with   ┆                                                │\n",
      "│ time bound rock solid results. The notion of    ┆                                                │\n",
      "│ six sigma was created by Motorola in 1986 as a  ┆                                                │\n",
      "│ result of consumers’ complaints about           ┆                                                │\n",
      "│ sub-standard Motorola phones’ quality. With the ┆                                                │\n",
      "│ passage of time, the philosophy was further     ┆                                                │\n",
      "│ refined by General Electric and was adopted     ┆                                                │\n",
      "│ worldwide by various manufacturing firms.       ┆                                                │\n",
      "│ Six Sigma Tip[s on Methodology says:            ┆                                                │\n",
      "│ “Any type of process which does not result in   ┆                                                │\n",
      "│ customer’s satisfaction is referred as a        ┆                                                │\n",
      "│ “Defect” and it has to be excluded from the     ┆                                                │\n",
      "│ process in order to get matchless supreme       ┆                                                │\n",
      "│ quality products and services”.                 ┆                                                │\n",
      "│ Six Sigma consists of further two sub-methods.  ┆                                                │\n",
      "│ These are:                                      ┆                                                │\n",
      "│ - DMAIC (Define, Measure, Analyze, Improve,     ┆                                                │\n",
      "│ Control)                                        ┆                                                │\n",
      "│ - DMADV (Design, Measure, Analyze, Design,      ┆                                                │\n",
      "│ Verify)                                         ┆                                                │\n",
      "│ Amazing DMAIC Methodology to Try Right Now!     ┆                                                │\n",
      "│ Get your all doubts vanish in few seconds by    ┆                                                │\n",
      "│ getting these Six Sigma tips and more insights  ┆                                                │\n",
      "│ about DMAIC methodology explained just right    ┆                                                │\n",
      "│ below:                                          ┆                                                │\n",
      "│ DMAIC has five steps which are defined step     ┆                                                │\n",
      "│ wise in sequence.                               ┆                                                │\n",
      "│ D – Define the Problem statement:               ┆                                                │\n",
      "│ Before trying your hands on to the problem, it  ┆                                                │\n",
      "│ is important to define the problem statement    ┆                                                │\n",
      "│ first. Follow 5W 1H rule to define your problem ┆                                                │\n",
      "│ in detail whereas 5Ws are What, When, Why,      ┆                                                │\n",
      "│ Where, Who and 1H is How.                       ┆                                                │\n",
      "│ M – Measure:                                    ┆                                                │\n",
      "│ After defining the problem statement in detail, ┆                                                │\n",
      "│ data collection is the next step. Compile all   ┆                                                │\n",
      "│ relevant data that provides you exact insight   ┆                                                │\n",
      "│ for the problem diagnosis.                      ┆                                                │\n",
      "│ A – Analyze:                                    ┆                                                │\n",
      "│ The data collection that was done in previous   ┆                                                │\n",
      "│ step will be analyzed thoroughly where root     ┆                                                │\n",
      "│ cause analysis will be done and ways to         ┆                                                │\n",
      "│ eradicate such defects will be looked into.     ┆                                                │\n",
      "│ I – Improve                                     ┆                                                │\n",
      "│ Process improvement will be carried out on the  ┆                                                │\n",
      "│ foundation of facts and figures derived from    ┆                                                │\n",
      "│ research done in previous steps. Efforts will   ┆                                                │\n",
      "│ be directed towards process improvement leading ┆                                                │\n",
      "│ to peerless quality outcomes.                   ┆                                                │\n",
      "│ C – Control                                     ┆                                                │\n",
      "│ Implement control measures to the processes     ┆                                                │\n",
      "│ that ensure defect free product every time with ┆                                                │\n",
      "│ superior precision and accuracy.                ┆                                                │\n",
      "│ How to Boost New Product Development by DMADV   ┆                                                │\n",
      "│ Method?                                         ┆                                                │\n",
      "│ Let’s unlock the potentials of DMADV method.    ┆                                                │\n",
      "│ D – Design                                      ┆                                                │\n",
      "│ In Design phase, one has to design the process  ┆                                                │\n",
      "│ from the scratch with all the relevant          ┆                                                │\n",
      "│ strategies that will give unbeatable results on ┆                                                │\n",
      "│ prototype small scale.                          ┆                                                │\n",
      "│ M – Measure                                     ┆                                                │\n",
      "│ Measure all the defined parameters that holds   ┆                                                │\n",
      "│ significance with respect to high quality       ┆                                                │\n",
      "│ delivery at all levels.                         ┆                                                │\n",
      "│ A – Analyze                                     ┆                                                │\n",
      "│ Analyze what has been measured in previous step ┆                                                │\n",
      "│ keeping design constraints in mind.             ┆                                                │\n",
      "│ D – Design                                      ┆                                                │\n",
      "│ Go for designing considering major and minor    ┆                                                │\n",
      "│ details of process on big scale.                ┆                                                │\n",
      "│ V – Verify                                      ┆                                                │\n",
      "│ Verify the process and its flow. Observe and    ┆                                                │\n",
      "│ record the results after successful             ┆                                                │\n",
      "│ implementation of the designed process.         ┆                                                │\n",
      "│ Getting Smart with Six Sigma Belt System        ┆                                                │\n",
      "│ As similar as Martial Arts system, Six Sigma    ┆                                                │\n",
      "│ also comes with Belt system with different      ┆                                                │\n",
      "│ level of authority system along with experience ┆                                                │\n",
      "│ and level of competency and trainings.          ┆                                                │\n",
      "│ Employees start with initial levels i.e. White  ┆                                                │\n",
      "│ Belt System and then employees can gradually    ┆                                                │\n",
      "│ increase their levels depending on their        ┆                                                │\n",
      "│ interest, experience and capabilities.          ┆                                                │\n",
      "│ Six Sigma White Belt System                     ┆                                                │\n",
      "│ Level: First Level of Belt System               ┆                                                │\n",
      "│ Knowledge Criteria: Mandatory to complete       ┆                                                │\n",
      "│ several hours of training about Six Sigma to    ┆                                                │\n",
      "│ get the basic clarity about the basics of it.   ┆                                                │\n",
      "│ Six Sigma Tips Yellow Belt                      ┆                                                │\n",
      "│ Level: Second level of Six Sigma                ┆                                                │\n",
      "│ Knowledge Criteria: Mandatory to complete 10 to ┆                                                │\n",
      "│ 15 hours Six Sigma classroom trainings to       ┆                                                │\n",
      "│ understand a bit advance steps methods.         ┆                                                │\n",
      "│ Six Sigma Tips: Green Belt                      ┆                                                │\n",
      "│ Level: Fundamental of success for any project.  ┆                                                │\n",
      "│ Knowledge Criteria: A classroom training        ┆                                                │\n",
      "│ session consisting for weeks is compulsory to   ┆                                                │\n",
      "│ attend and have to pass a written exam along    ┆                                                │\n",
      "│ with serving in a Six Sigma project team.       ┆                                                │\n",
      "│ Six Sigma Tips: Black Belt                      ┆                                                │\n",
      "│ Level: Full time job for Black Belt. Advance    ┆                                                │\n",
      "│ level of Six Sigma.                             ┆                                                │\n",
      "│ Knowledge Criteria: To become a certified Six   ┆                                                │\n",
      "│ Sigma Black Belt, candidates must pass a        ┆                                                │\n",
      "│ written exam and successfully complete          ┆                                                │\n",
      "│ projects.                                       ┆                                                │\n",
      "│ Six Sigma Tips: Master Black Belt               ┆                                                │\n",
      "│ Level: Highest Six Sigma Belt ranking and the   ┆                                                │\n",
      "│ certification is in high demand for customers.  ┆                                                │\n",
      "│ Knowledge Criteria: To become Six Sigma Master  ┆                                                │\n",
      "│ Black Belt, one has to opt an experience of at  ┆                                                │\n",
      "│ least five years as Black Belt along with       ┆                                                │\n",
      "│ successful completion of minimum of ten six     ┆                                                │\n",
      "│ sigma projects. The role of Master Black Belts  ┆                                                │\n",
      "│ is not only to ensure active participation and  ┆                                                │\n",
      "│ efficient problem solving in six sigma projects ┆                                                │\n",
      "│ but to trickle down six sigma culture and       ┆                                                │\n",
      "│ nourish six sigma techniques among employees is ┆                                                │\n",
      "│ one of the objective of Six Sigma Master Black  ┆                                                │\n",
      "│ Belt.                                           ┆                                                │\n",
      "│ What Not Everybody Ought to Know About Six      ┆                                                │\n",
      "│ Sigma                                           ┆                                                │\n",
      "│ Do you wish more people bought your product, or ┆                                                │\n",
      "│ your sales to be sky-rocketed without investing ┆                                                │\n",
      "│ years and years? Are you still wasting your     ┆                                                │\n",
      "│ time on reoccurring problems daily? Let us      ┆                                                │\n",
      "│ unlock few of the many benefits of Six Sigma    ┆                                                │\n",
      "│ which your consultant will never tell you       ┆                                                │\n",
      "│ about.                                          ┆                                                │\n",
      "│ - Proven Way to Enhance Customer Retention:     ┆                                                │\n",
      "│ A single dissatisfied customer can take your    ┆                                                │\n",
      "│ millions of business away and on the other hand ┆                                                │\n",
      "│ one loyal customer can bring in the             ┆                                                │\n",
      "│ opportunities to invite other new customers as  ┆                                                │\n",
      "│ well so to retain your regular customers should ┆                                                │\n",
      "│ be your top most priority by all means.         ┆                                                │\n",
      "│ - How to be able to Achieve Targets in an Easy  ┆                                                │\n",
      "│ Way?                                            ┆                                                │\n",
      "│ Set SMART goals to accomplish anything whereas  ┆                                                │\n",
      "│ the term SMART is an abbreviation whose full    ┆                                                │\n",
      "│ form is Specific, Measurable, Achievable,       ┆                                                │\n",
      "│ Relevant, Timebound. Before setting SMART       ┆                                                │\n",
      "│ objectives, it is important to know every       ┆                                                │\n",
      "│ employees’ training needs, learning agility and ┆                                                │\n",
      "│ performance curve accordingly.                  ┆                                                │\n",
      "│ - The Best Ever Planning Solution for Any       ┆                                                │\n",
      "│ Business                                        ┆                                                │\n",
      "│ Six Sigma tips help in great ways to get rid of ┆                                                │\n",
      "│ distractions and to maintain your focus in      ┆                                                │\n",
      "│ unidirectional way. SWOT (Strength, Weakness,   ┆                                                │\n",
      "│ Opportunities and Threats) analysis remains     ┆                                                │\n",
      "│ your eyes to the areas of improvements which    ┆                                                │\n",
      "│ eventually can turn your business a worth more  ┆                                                │\n",
      "│ than the present situation.                     ┆                                                │\n",
      "│ - How Big Firms Are Able To Reduce Their Cycle  ┆                                                │\n",
      "│ Times?                                          ┆                                                │\n",
      "│ According to numerous business reports, 35%     ┆                                                │\n",
      "│ reduction in cycle times have been observed     ┆                                                │\n",
      "│ after implementation of Six Sigma. On the       ┆                                                │\n",
      "│ contrary, firms who haven’t implemented six     ┆                                                │\n",
      "│ sigma methodology complaint about delays in     ┆                                                │\n",
      "│ their projects and cycle time completion, less  ┆                                                │\n",
      "│ employee engagement or totally zero employee    ┆                                                │\n",
      "│ motivational levels.                            ┆                                                │\n",
      "│ Real Life Amazing Results of Six Sigma: 3M      ┆                                                │\n",
      "│ Success Story                                   ┆                                                │\n",
      "│ 3M reported to have been celebrated their       ┆                                                │\n",
      "│ success credited by Six Sigma back in early     ┆                                                │\n",
      "│ 2000’s. Spare few minutes to have a glance on   ┆                                                │\n",
      "│ impressive ground breaking results made real    ┆                                                │\n",
      "│ only through Six Sigma.                         ┆                                                │\n",
      "│ - 50% reduction in waste generation             ┆                                                │\n",
      "│ - 67% reduction in GreenHouse Gas (GHG)         ┆                                                │\n",
      "│ emissions                                       ┆                                                │\n",
      "│ - 37% water recycles usage                      ┆                                                │\n",
      "│ - 8% reduction in toxic air emissions           ┆                                                │\n",
      "│ If Six Sigma tips can do wonders for 3M then    ┆                                                │\n",
      "│ why not you? Let’s banish the hindrances and    ┆                                                │\n",
      "│ fears in your way to become world class         ┆                                                │\n",
      "│ business.                                       ┆                                                │\n",
      "│ Israeli night raids targeting arms depots and   ┆ <urn:uuid:c6c18330-fd72-4035-a16d-b91f69ef3085 │\n",
      "│ military positions in eastern Syria killed at   ┆ >                                              │\n",
      "│ least five soldiers and 11 allied fighters, the ┆                                                │\n",
      "│ Syrian Observatory for Human Rights said on     ┆                                                │\n",
      "│ Wednesday.                                      ┆                                                │\n",
      "│ The Israeli air force carried out more than 18  ┆                                                │\n",
      "│ strikes against multiple targets in an area     ┆                                                │\n",
      "│ stretching from the eastern town of Deir ez-Zor ┆                                                │\n",
      "│ to the Boukamal desert at the Syrian-Iraqi      ┆                                                │\n",
      "│ border, according to the Britain-based war      ┆                                                │\n",
      "│ monitor.                                        ┆                                                │\n",
      "│ The raids killed five Syrian soldiers and 11    ┆                                                │\n",
      "│ allied fighters belonging to the Iranian        ┆                                                │\n",
      "│ Revolutionary Guards, Lebanese Hezbollah and    ┆                                                │\n",
      "│ the Fatimid Brigade, which includes pro-Iranian ┆                                                │\n",
      "│ Afghan fighters, the Observatory said, although ┆                                                │\n",
      "│ their nationalities and a precise breakdown     ┆                                                │\n",
      "│ were not immediately known.                     ┆                                                │\n",
      "│ The Syrian state news agency SANA reported the  ┆                                                │\n",
      "│ strikes but without giving further details.     ┆                                                │\n",
      "│ “At 1:10 am, the Israeli enemy carried out an   ┆                                                │\n",
      "│ aerial assault on the town of Deir ez-Zor and   ┆                                                │\n",
      "│ the Boukamal region,” SANA said, citing a       ┆                                                │\n",
      "│ military source.                                ┆                                                │\n",
      "│ “The results of the aggression are currently    ┆                                                │\n",
      "│ being verified,” it added.                      ┆                                                │\n",
      "│ It was the second wave of Israeli raids in      ┆                                                │\n",
      "│ Syria in less than a week.                      ┆                                                │\n",
      "│ The last strikes on January 7 targeted          ┆                                                │\n",
      "│ positions in southern Syria and south of the    ┆                                                │\n",
      "│ capital Damascus, killing three pro-Iran        ┆                                                │\n",
      "│ fighters.                                       ┆                                                │\n",
      "│ Israel routinely carries out raids in Syria,    ┆                                                │\n",
      "│ mostly against targets affiliated with Iran in  ┆                                                │\n",
      "│ what it says is a bid to prevent its arch foe   ┆                                                │\n",
      "│ from securing further foothold along its        ┆                                                │\n",
      "│ borders.                                        ┆                                                │\n",
      "│ Iran has members of its own military as well as ┆                                                │\n",
      "│ fighters from a variety of nationalities        ┆                                                │\n",
      "│ fighting with militias it supports deployed     ┆                                                │\n",
      "│ across Syria.                                   ┆                                                │\n",
      "│ Israel hit around 50 targets in Syria in 2020,  ┆                                                │\n",
      "│ according to an annual report released in late  ┆                                                │\n",
      "│ December by the Israeli military.               ┆                                                │\n",
      "│ The Israeli army has carried out hundreds of    ┆                                                │\n",
      "│ air and missile strikes on Syria since the      ┆                                                │\n",
      "│ civil war broke out in 2011, targeting Iranian  ┆                                                │\n",
      "│ and Lebanese Hezbollah forces as well as        ┆                                                │\n",
      "│ government troops.                              ┆                                                │\n",
      "│ The Jewish state rarely acknowledges individual ┆                                                │\n",
      "│ strikes.                                        ┆                                                │\n",
      "│ The Syrian Observer has not verified the        ┆                                                │\n",
      "│ content of this story. Responsibility for the   ┆                                                │\n",
      "│ information and views set out in this article   ┆                                                │\n",
      "│ lies entirely with the author.                  ┆                                                │\n",
      "└─────────────────────────────────────────────────┴────────────────────────────────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "import polars as pl\n",
    "import os\n",
    "input_df = pl.read_parquet(os.path.join(os.path.abspath(\"\"), \"test-data\", \"input\", \"readability-test.parquet\"))\n",
    "\n",
    "with pl.Config(fmt_str_lengths=10000000, tbl_rows=-1, tbl_cols=-1):\n",
    "    print(input_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5305d127-10fd-4fa6-97a6-ac47db2bdc7e",
   "metadata": {},
   "source": [
    "***** print the output result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0b2eddb9-4fb6-41eb-916c-3741b9129f2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (2, 3)\n",
      "┌────────────────────────────────────┬───────────────────────────────────┬─────────────────────────┐\n",
      "│ contents                           ┆ id                                ┆ mcalpine_eflaw_textstat │\n",
      "│ ---                                ┆ ---                               ┆ ---                     │\n",
      "│ str                                ┆ str                               ┆ f64                     │\n",
      "╞════════════════════════════════════╪═══════════════════════════════════╪═════════════════════════╡\n",
      "│ Six Sigma Tips                     ┆ <urn:uuid:01d4d837-d379-4466-8715 ┆ 28.7                    │\n",
      "│ These Six Sigma tips help you      ┆ -3d1edda758c1>                    ┆                         │\n",
      "│ achieve high quality every time,   ┆                                   ┆                         │\n",
      "│ the desired objective of most      ┆                                   ┆                         │\n",
      "│ every business. However to get the ┆                                   ┆                         │\n",
      "│ desire of high quality creations   ┆                                   ┆                         │\n",
      "│ turn into reality calls to extend  ┆                                   ┆                         │\n",
      "│ yourself for some extra miles. Six ┆                                   ┆                         │\n",
      "│ Sigma is a success proven business ┆                                   ┆                         │\n",
      "│ management scheme that can assist  ┆                                   ┆                         │\n",
      "│ you to unleash new horizons of     ┆                                   ┆                         │\n",
      "│ business success with time bound   ┆                                   ┆                         │\n",
      "│ rock solid results. The notion of  ┆                                   ┆                         │\n",
      "│ six sigma was created by Motorola  ┆                                   ┆                         │\n",
      "│ in 1986 as a result of consumers’  ┆                                   ┆                         │\n",
      "│ complaints about sub-standard      ┆                                   ┆                         │\n",
      "│ Motorola phones’ quality. With the ┆                                   ┆                         │\n",
      "│ passage of time, the philosophy    ┆                                   ┆                         │\n",
      "│ was further refined by General     ┆                                   ┆                         │\n",
      "│ Electric and was adopted worldwide ┆                                   ┆                         │\n",
      "│ by various manufacturing firms.    ┆                                   ┆                         │\n",
      "│ Six Sigma Tip[s on Methodology     ┆                                   ┆                         │\n",
      "│ says:                              ┆                                   ┆                         │\n",
      "│ “Any type of process which does    ┆                                   ┆                         │\n",
      "│ not result in customer’s           ┆                                   ┆                         │\n",
      "│ satisfaction is referred as a      ┆                                   ┆                         │\n",
      "│ “Defect” and it has to be excluded ┆                                   ┆                         │\n",
      "│ from the process in order to get   ┆                                   ┆                         │\n",
      "│ matchless supreme quality products ┆                                   ┆                         │\n",
      "│ and services”.                     ┆                                   ┆                         │\n",
      "│ Six Sigma consists of further two  ┆                                   ┆                         │\n",
      "│ sub-methods. These are:            ┆                                   ┆                         │\n",
      "│ - DMAIC (Define, Measure, Analyze, ┆                                   ┆                         │\n",
      "│ Improve, Control)                  ┆                                   ┆                         │\n",
      "│ - DMADV (Design, Measure, Analyze, ┆                                   ┆                         │\n",
      "│ Design, Verify)                    ┆                                   ┆                         │\n",
      "│ Amazing DMAIC Methodology to Try   ┆                                   ┆                         │\n",
      "│ Right Now!                         ┆                                   ┆                         │\n",
      "│ Get your all doubts vanish in few  ┆                                   ┆                         │\n",
      "│ seconds by getting these Six Sigma ┆                                   ┆                         │\n",
      "│ tips and more insights about DMAIC ┆                                   ┆                         │\n",
      "│ methodology explained just right   ┆                                   ┆                         │\n",
      "│ below:                             ┆                                   ┆                         │\n",
      "│ DMAIC has five steps which are     ┆                                   ┆                         │\n",
      "│ defined step wise in sequence.     ┆                                   ┆                         │\n",
      "│ D – Define the Problem statement:  ┆                                   ┆                         │\n",
      "│ Before trying your hands on to the ┆                                   ┆                         │\n",
      "│ problem, it is important to define ┆                                   ┆                         │\n",
      "│ the problem statement first.       ┆                                   ┆                         │\n",
      "│ Follow 5W 1H rule to define your   ┆                                   ┆                         │\n",
      "│ problem in detail whereas 5Ws are  ┆                                   ┆                         │\n",
      "│ What, When, Why, Where, Who and 1H ┆                                   ┆                         │\n",
      "│ is How.                            ┆                                   ┆                         │\n",
      "│ M – Measure:                       ┆                                   ┆                         │\n",
      "│ After defining the problem         ┆                                   ┆                         │\n",
      "│ statement in detail, data          ┆                                   ┆                         │\n",
      "│ collection is the next step.       ┆                                   ┆                         │\n",
      "│ Compile all relevant data that     ┆                                   ┆                         │\n",
      "│ provides you exact insight for the ┆                                   ┆                         │\n",
      "│ problem diagnosis.                 ┆                                   ┆                         │\n",
      "│ A – Analyze:                       ┆                                   ┆                         │\n",
      "│ The data collection that was done  ┆                                   ┆                         │\n",
      "│ in previous step will be analyzed  ┆                                   ┆                         │\n",
      "│ thoroughly where root cause        ┆                                   ┆                         │\n",
      "│ analysis will be done and ways to  ┆                                   ┆                         │\n",
      "│ eradicate such defects will be     ┆                                   ┆                         │\n",
      "│ looked into.                       ┆                                   ┆                         │\n",
      "│ I – Improve                        ┆                                   ┆                         │\n",
      "│ Process improvement will be        ┆                                   ┆                         │\n",
      "│ carried out on the foundation of   ┆                                   ┆                         │\n",
      "│ facts and figures derived from     ┆                                   ┆                         │\n",
      "│ research done in previous steps.   ┆                                   ┆                         │\n",
      "│ Efforts will be directed towards   ┆                                   ┆                         │\n",
      "│ process improvement leading to     ┆                                   ┆                         │\n",
      "│ peerless quality outcomes.         ┆                                   ┆                         │\n",
      "│ C – Control                        ┆                                   ┆                         │\n",
      "│ Implement control measures to the  ┆                                   ┆                         │\n",
      "│ processes that ensure defect free  ┆                                   ┆                         │\n",
      "│ product every time with superior   ┆                                   ┆                         │\n",
      "│ precision and accuracy.            ┆                                   ┆                         │\n",
      "│ How to Boost New Product           ┆                                   ┆                         │\n",
      "│ Development by DMADV Method?       ┆                                   ┆                         │\n",
      "│ Let’s unlock the potentials of     ┆                                   ┆                         │\n",
      "│ DMADV method.                      ┆                                   ┆                         │\n",
      "│ D – Design                         ┆                                   ┆                         │\n",
      "│ In Design phase, one has to design ┆                                   ┆                         │\n",
      "│ the process from the scratch with  ┆                                   ┆                         │\n",
      "│ all the relevant strategies that   ┆                                   ┆                         │\n",
      "│ will give unbeatable results on    ┆                                   ┆                         │\n",
      "│ prototype small scale.             ┆                                   ┆                         │\n",
      "│ M – Measure                        ┆                                   ┆                         │\n",
      "│ Measure all the defined parameters ┆                                   ┆                         │\n",
      "│ that holds significance with       ┆                                   ┆                         │\n",
      "│ respect to high quality delivery   ┆                                   ┆                         │\n",
      "│ at all levels.                     ┆                                   ┆                         │\n",
      "│ A – Analyze                        ┆                                   ┆                         │\n",
      "│ Analyze what has been measured in  ┆                                   ┆                         │\n",
      "│ previous step keeping design       ┆                                   ┆                         │\n",
      "│ constraints in mind.               ┆                                   ┆                         │\n",
      "│ D – Design                         ┆                                   ┆                         │\n",
      "│ Go for designing considering major ┆                                   ┆                         │\n",
      "│ and minor details of process on    ┆                                   ┆                         │\n",
      "│ big scale.                         ┆                                   ┆                         │\n",
      "│ V – Verify                         ┆                                   ┆                         │\n",
      "│ Verify the process and its flow.   ┆                                   ┆                         │\n",
      "│ Observe and record the results     ┆                                   ┆                         │\n",
      "│ after successful implementation of ┆                                   ┆                         │\n",
      "│ the designed process.              ┆                                   ┆                         │\n",
      "│ Getting Smart with Six Sigma Belt  ┆                                   ┆                         │\n",
      "│ System                             ┆                                   ┆                         │\n",
      "│ As similar as Martial Arts system, ┆                                   ┆                         │\n",
      "│ Six Sigma also comes with Belt     ┆                                   ┆                         │\n",
      "│ system with different level of     ┆                                   ┆                         │\n",
      "│ authority system along with        ┆                                   ┆                         │\n",
      "│ experience and level of competency ┆                                   ┆                         │\n",
      "│ and trainings. Employees start     ┆                                   ┆                         │\n",
      "│ with initial levels i.e. White     ┆                                   ┆                         │\n",
      "│ Belt System and then employees can ┆                                   ┆                         │\n",
      "│ gradually increase their levels    ┆                                   ┆                         │\n",
      "│ depending on their interest,       ┆                                   ┆                         │\n",
      "│ experience and capabilities.       ┆                                   ┆                         │\n",
      "│ Six Sigma White Belt System        ┆                                   ┆                         │\n",
      "│ Level: First Level of Belt System  ┆                                   ┆                         │\n",
      "│ Knowledge Criteria: Mandatory to   ┆                                   ┆                         │\n",
      "│ complete several hours of training ┆                                   ┆                         │\n",
      "│ about Six Sigma to get the basic   ┆                                   ┆                         │\n",
      "│ clarity about the basics of it.    ┆                                   ┆                         │\n",
      "│ Six Sigma Tips Yellow Belt         ┆                                   ┆                         │\n",
      "│ Level: Second level of Six Sigma   ┆                                   ┆                         │\n",
      "│ Knowledge Criteria: Mandatory to   ┆                                   ┆                         │\n",
      "│ complete 10 to 15 hours Six Sigma  ┆                                   ┆                         │\n",
      "│ classroom trainings to understand  ┆                                   ┆                         │\n",
      "│ a bit advance steps methods.       ┆                                   ┆                         │\n",
      "│ Six Sigma Tips: Green Belt         ┆                                   ┆                         │\n",
      "│ Level: Fundamental of success for  ┆                                   ┆                         │\n",
      "│ any project.                       ┆                                   ┆                         │\n",
      "│ Knowledge Criteria: A classroom    ┆                                   ┆                         │\n",
      "│ training session consisting for    ┆                                   ┆                         │\n",
      "│ weeks is compulsory to attend and  ┆                                   ┆                         │\n",
      "│ have to pass a written exam along  ┆                                   ┆                         │\n",
      "│ with serving in a Six Sigma        ┆                                   ┆                         │\n",
      "│ project team.                      ┆                                   ┆                         │\n",
      "│ Six Sigma Tips: Black Belt         ┆                                   ┆                         │\n",
      "│ Level: Full time job for Black     ┆                                   ┆                         │\n",
      "│ Belt. Advance level of Six Sigma.  ┆                                   ┆                         │\n",
      "│ Knowledge Criteria: To become a    ┆                                   ┆                         │\n",
      "│ certified Six Sigma Black Belt,    ┆                                   ┆                         │\n",
      "│ candidates must pass a written     ┆                                   ┆                         │\n",
      "│ exam and successfully complete     ┆                                   ┆                         │\n",
      "│ projects.                          ┆                                   ┆                         │\n",
      "│ Six Sigma Tips: Master Black Belt  ┆                                   ┆                         │\n",
      "│ Level: Highest Six Sigma Belt      ┆                                   ┆                         │\n",
      "│ ranking and the certification is   ┆                                   ┆                         │\n",
      "│ in high demand for customers.      ┆                                   ┆                         │\n",
      "│ Knowledge Criteria: To become Six  ┆                                   ┆                         │\n",
      "│ Sigma Master Black Belt, one has   ┆                                   ┆                         │\n",
      "│ to opt an experience of at least   ┆                                   ┆                         │\n",
      "│ five years as Black Belt along     ┆                                   ┆                         │\n",
      "│ with successful completion of      ┆                                   ┆                         │\n",
      "│ minimum of ten six sigma projects. ┆                                   ┆                         │\n",
      "│ The role of Master Black Belts is  ┆                                   ┆                         │\n",
      "│ not only to ensure active          ┆                                   ┆                         │\n",
      "│ participation and efficient        ┆                                   ┆                         │\n",
      "│ problem solving in six sigma       ┆                                   ┆                         │\n",
      "│ projects but to trickle down six   ┆                                   ┆                         │\n",
      "│ sigma culture and nourish six      ┆                                   ┆                         │\n",
      "│ sigma techniques among employees   ┆                                   ┆                         │\n",
      "│ is one of the objective of Six     ┆                                   ┆                         │\n",
      "│ Sigma Master Black Belt.           ┆                                   ┆                         │\n",
      "│ What Not Everybody Ought to Know   ┆                                   ┆                         │\n",
      "│ About Six Sigma                    ┆                                   ┆                         │\n",
      "│ Do you wish more people bought     ┆                                   ┆                         │\n",
      "│ your product, or your sales to be  ┆                                   ┆                         │\n",
      "│ sky-rocketed without investing     ┆                                   ┆                         │\n",
      "│ years and years? Are you still     ┆                                   ┆                         │\n",
      "│ wasting your time on reoccurring   ┆                                   ┆                         │\n",
      "│ problems daily? Let us unlock few  ┆                                   ┆                         │\n",
      "│ of the many benefits of Six Sigma  ┆                                   ┆                         │\n",
      "│ which your consultant will never   ┆                                   ┆                         │\n",
      "│ tell you about.                    ┆                                   ┆                         │\n",
      "│ - Proven Way to Enhance Customer   ┆                                   ┆                         │\n",
      "│ Retention:                         ┆                                   ┆                         │\n",
      "│ A single dissatisfied customer can ┆                                   ┆                         │\n",
      "│ take your millions of business     ┆                                   ┆                         │\n",
      "│ away and on the other hand one     ┆                                   ┆                         │\n",
      "│ loyal customer can bring in the    ┆                                   ┆                         │\n",
      "│ opportunities to invite other new  ┆                                   ┆                         │\n",
      "│ customers as well so to retain     ┆                                   ┆                         │\n",
      "│ your regular customers should be   ┆                                   ┆                         │\n",
      "│ your top most priority by all      ┆                                   ┆                         │\n",
      "│ means.                             ┆                                   ┆                         │\n",
      "│ - How to be able to Achieve        ┆                                   ┆                         │\n",
      "│ Targets in an Easy Way?            ┆                                   ┆                         │\n",
      "│ Set SMART goals to accomplish      ┆                                   ┆                         │\n",
      "│ anything whereas the term SMART is ┆                                   ┆                         │\n",
      "│ an abbreviation whose full form is ┆                                   ┆                         │\n",
      "│ Specific, Measurable, Achievable,  ┆                                   ┆                         │\n",
      "│ Relevant, Timebound. Before        ┆                                   ┆                         │\n",
      "│ setting SMART objectives, it is    ┆                                   ┆                         │\n",
      "│ important to know every employees’ ┆                                   ┆                         │\n",
      "│ training needs, learning agility   ┆                                   ┆                         │\n",
      "│ and performance curve accordingly. ┆                                   ┆                         │\n",
      "│ - The Best Ever Planning Solution  ┆                                   ┆                         │\n",
      "│ for Any Business                   ┆                                   ┆                         │\n",
      "│ Six Sigma tips help in great ways  ┆                                   ┆                         │\n",
      "│ to get rid of distractions and to  ┆                                   ┆                         │\n",
      "│ maintain your focus in             ┆                                   ┆                         │\n",
      "│ unidirectional way. SWOT           ┆                                   ┆                         │\n",
      "│ (Strength, Weakness, Opportunities ┆                                   ┆                         │\n",
      "│ and Threats) analysis remains your ┆                                   ┆                         │\n",
      "│ eyes to the areas of improvements  ┆                                   ┆                         │\n",
      "│ which eventually can turn your     ┆                                   ┆                         │\n",
      "│ business a worth more than the     ┆                                   ┆                         │\n",
      "│ present situation.                 ┆                                   ┆                         │\n",
      "│ - How Big Firms Are Able To Reduce ┆                                   ┆                         │\n",
      "│ Their Cycle Times?                 ┆                                   ┆                         │\n",
      "│ According to numerous business     ┆                                   ┆                         │\n",
      "│ reports, 35% reduction in cycle    ┆                                   ┆                         │\n",
      "│ times have been observed after     ┆                                   ┆                         │\n",
      "│ implementation of Six Sigma. On    ┆                                   ┆                         │\n",
      "│ the contrary, firms who haven’t    ┆                                   ┆                         │\n",
      "│ implemented six sigma methodology  ┆                                   ┆                         │\n",
      "│ complaint about delays in their    ┆                                   ┆                         │\n",
      "│ projects and cycle time            ┆                                   ┆                         │\n",
      "│ completion, less employee          ┆                                   ┆                         │\n",
      "│ engagement or totally zero         ┆                                   ┆                         │\n",
      "│ employee motivational levels.      ┆                                   ┆                         │\n",
      "│ Real Life Amazing Results of Six   ┆                                   ┆                         │\n",
      "│ Sigma: 3M Success Story            ┆                                   ┆                         │\n",
      "│ 3M reported to have been           ┆                                   ┆                         │\n",
      "│ celebrated their success credited  ┆                                   ┆                         │\n",
      "│ by Six Sigma back in early 2000’s. ┆                                   ┆                         │\n",
      "│ Spare few minutes to have a glance ┆                                   ┆                         │\n",
      "│ on impressive ground breaking      ┆                                   ┆                         │\n",
      "│ results made real only through Six ┆                                   ┆                         │\n",
      "│ Sigma.                             ┆                                   ┆                         │\n",
      "│ - 50% reduction in waste           ┆                                   ┆                         │\n",
      "│ generation                         ┆                                   ┆                         │\n",
      "│ - 67% reduction in GreenHouse Gas  ┆                                   ┆                         │\n",
      "│ (GHG) emissions                    ┆                                   ┆                         │\n",
      "│ - 37% water recycles usage         ┆                                   ┆                         │\n",
      "│ - 8% reduction in toxic air        ┆                                   ┆                         │\n",
      "│ emissions                          ┆                                   ┆                         │\n",
      "│ If Six Sigma tips can do wonders   ┆                                   ┆                         │\n",
      "│ for 3M then why not you? Let’s     ┆                                   ┆                         │\n",
      "│ banish the hindrances and fears in ┆                                   ┆                         │\n",
      "│ your way to become world class     ┆                                   ┆                         │\n",
      "│ business.                          ┆                                   ┆                         │\n",
      "│ Israeli night raids targeting arms ┆ <urn:uuid:c6c18330-fd72-4035-a16d ┆ 31.1                    │\n",
      "│ depots and military positions in   ┆ -b91f69ef3085>                    ┆                         │\n",
      "│ eastern Syria killed at least five ┆                                   ┆                         │\n",
      "│ soldiers and 11 allied fighters,   ┆                                   ┆                         │\n",
      "│ the Syrian Observatory for Human   ┆                                   ┆                         │\n",
      "│ Rights said on Wednesday.          ┆                                   ┆                         │\n",
      "│ The Israeli air force carried out  ┆                                   ┆                         │\n",
      "│ more than 18 strikes against       ┆                                   ┆                         │\n",
      "│ multiple targets in an area        ┆                                   ┆                         │\n",
      "│ stretching from the eastern town   ┆                                   ┆                         │\n",
      "│ of Deir ez-Zor to the Boukamal     ┆                                   ┆                         │\n",
      "│ desert at the Syrian-Iraqi border, ┆                                   ┆                         │\n",
      "│ according to the Britain-based war ┆                                   ┆                         │\n",
      "│ monitor.                           ┆                                   ┆                         │\n",
      "│ The raids killed five Syrian       ┆                                   ┆                         │\n",
      "│ soldiers and 11 allied fighters    ┆                                   ┆                         │\n",
      "│ belonging to the Iranian           ┆                                   ┆                         │\n",
      "│ Revolutionary Guards, Lebanese     ┆                                   ┆                         │\n",
      "│ Hezbollah and the Fatimid Brigade, ┆                                   ┆                         │\n",
      "│ which includes pro-Iranian Afghan  ┆                                   ┆                         │\n",
      "│ fighters, the Observatory said,    ┆                                   ┆                         │\n",
      "│ although their nationalities and a ┆                                   ┆                         │\n",
      "│ precise breakdown were not         ┆                                   ┆                         │\n",
      "│ immediately known.                 ┆                                   ┆                         │\n",
      "│ The Syrian state news agency SANA  ┆                                   ┆                         │\n",
      "│ reported the strikes but without   ┆                                   ┆                         │\n",
      "│ giving further details.            ┆                                   ┆                         │\n",
      "│ “At 1:10 am, the Israeli enemy     ┆                                   ┆                         │\n",
      "│ carried out an aerial assault on   ┆                                   ┆                         │\n",
      "│ the town of Deir ez-Zor and the    ┆                                   ┆                         │\n",
      "│ Boukamal region,” SANA said,       ┆                                   ┆                         │\n",
      "│ citing a military source.          ┆                                   ┆                         │\n",
      "│ “The results of the aggression are ┆                                   ┆                         │\n",
      "│ currently being verified,” it      ┆                                   ┆                         │\n",
      "│ added.                             ┆                                   ┆                         │\n",
      "│ It was the second wave of Israeli  ┆                                   ┆                         │\n",
      "│ raids in Syria in less than a      ┆                                   ┆                         │\n",
      "│ week.                              ┆                                   ┆                         │\n",
      "│ The last strikes on January 7      ┆                                   ┆                         │\n",
      "│ targeted positions in southern     ┆                                   ┆                         │\n",
      "│ Syria and south of the capital     ┆                                   ┆                         │\n",
      "│ Damascus, killing three pro-Iran   ┆                                   ┆                         │\n",
      "│ fighters.                          ┆                                   ┆                         │\n",
      "│ Israel routinely carries out raids ┆                                   ┆                         │\n",
      "│ in Syria, mostly against targets   ┆                                   ┆                         │\n",
      "│ affiliated with Iran in what it    ┆                                   ┆                         │\n",
      "│ says is a bid to prevent its arch  ┆                                   ┆                         │\n",
      "│ foe from securing further foothold ┆                                   ┆                         │\n",
      "│ along its borders.                 ┆                                   ┆                         │\n",
      "│ Iran has members of its own        ┆                                   ┆                         │\n",
      "│ military as well as fighters from  ┆                                   ┆                         │\n",
      "│ a variety of nationalities         ┆                                   ┆                         │\n",
      "│ fighting with militias it supports ┆                                   ┆                         │\n",
      "│ deployed across Syria.             ┆                                   ┆                         │\n",
      "│ Israel hit around 50 targets in    ┆                                   ┆                         │\n",
      "│ Syria in 2020, according to an     ┆                                   ┆                         │\n",
      "│ annual report released in late     ┆                                   ┆                         │\n",
      "│ December by the Israeli military.  ┆                                   ┆                         │\n",
      "│ The Israeli army has carried out   ┆                                   ┆                         │\n",
      "│ hundreds of air and missile        ┆                                   ┆                         │\n",
      "│ strikes on Syria since the civil   ┆                                   ┆                         │\n",
      "│ war broke out in 2011, targeting   ┆                                   ┆                         │\n",
      "│ Iranian and Lebanese Hezbollah     ┆                                   ┆                         │\n",
      "│ forces as well as government       ┆                                   ┆                         │\n",
      "│ troops.                            ┆                                   ┆                         │\n",
      "│ The Jewish state rarely            ┆                                   ┆                         │\n",
      "│ acknowledges individual strikes.   ┆                                   ┆                         │\n",
      "│ The Syrian Observer has not        ┆                                   ┆                         │\n",
      "│ verified the content of this       ┆                                   ┆                         │\n",
      "│ story. Responsibility for the      ┆                                   ┆                         │\n",
      "│ information and views set out in   ┆                                   ┆                         │\n",
      "│ this article lies entirely with    ┆                                   ┆                         │\n",
      "│ the author.                        ┆                                   ┆                         │\n",
      "└────────────────────────────────────┴───────────────────────────────────┴─────────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "import polars as pl\n",
    "output_df = pl.read_parquet(os.path.join(os.path.abspath(\"\"), \"output\", \"readability-test.parquet\"))\n",
    "with pl.Config(fmt_str_lengths=10000000, tbl_rows=-1, tbl_cols=-1):\n",
    "    print(output_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d60e391d-cf58-47ae-9991-04c05d114edc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "787c644e-2640-4c05-bdc2-8a261305a89f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
